fix(native-eval): benchmark genuine OpenClaw code mode by vincentkoc · Pull Request #62 · openclaw/shellbench

vincentkoc · 2026-07-29T20:31:46Z

What does this PR do?

Makes the OpenClaw native-eval arms represent genuine direct, directory, and
Code Mode execution, then exports and validates their public trajectory bundles
before a run can be scored.

Why?

Fixes #61.

The earlier code arm selected the legacy tool_search_code bridge rather than
OpenClaw Code Mode. The harness also masked setup failures, forced thinking off,
terminated the runtime before lifecycle cleanup settled, and reconstructed
delegated traces from private session files. Those failures made the released
direct/code comparison invalid and disproportionately erased Code Mode's nested
tool calls.

Changes

Map direct, directory, and code to explicit, mutually exclusive
OpenClaw tool surfaces.
Propagate the requested reasoning effort through planning, dispatch,
OpenClaw defaults, subagents, CLI execution, and manifests.
Fail setup immediately and let openclaw agent exit naturally.
Export root and child trajectories through
openclaw sessions export-trajectory.
Capture child runs at the canonical pre-cleanup terminal hook, including
nested and repeated runs, without blocking the Gateway.
Validate audit initialization, trace/session identity, event counts, terminal
status, complete Code Mode snapshots, and the exact provider-visible
exec/wait surface.
Reconstruct Code Mode nested calls and complete session-tree usage from the
public export bundles.
Bound child-export stabilization and reject missing or failed evidence rather
than publishing partial traces.

Tests

Blacksmith Testbox full suite: 454 passed, 5 skipped
Blacksmith Testbox focused runner suite after final fixes: 61 passed
Ruff clean
Python compile, git diff --check, generated shell syntax for all three
modes, and generated audit-plugin node --check
Fresh Codex autoreview: clean
Fresh matched direct/Code Mode ShellBench batch

The July 29, 2026 released code result remains legacy Tool Search bridge
evidence only. It is not evidence about genuine OpenClaw Code Mode.

clawsweeper · 2026-07-29T20:36:00Z

Codex review: needs maintainer review before merge. Reviewed August 4, 2026, 8:19 PM ET / August 5, 2026, 00:19 UTC.

ClawSweeper review

What this changes

The PR makes ShellBench’s OpenClaw native-evaluation modes explicit, carries reasoning and mode settings through dispatch, and validates public trajectory exports before scoring.

Regression provenance

Possible regression — probable (reviewed change; known regression link). No predecessor PR is attributed.

Merge readiness

⚠️ Ready for maintainer review - 1 item remains

Keep this member-authored PR open for normal maintainer review. It corrects a source-proven benchmark-control defect, and the prior retired-input concern does not apply because those inputs are absent from current main.

Priority: P2
Reviewed head: f5b31cb1260b631b811bf0ede9f4eba51a94b98c

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	The patch is coherent, has substantial focused coverage, and has no remaining discrete correctness finding.
Proof confidence	🌊 off-meta tidepool	Not applicable: This member-authored PR reports Blacksmith Testbox full-suite and focused-run results; the external-contributor proof gate does not apply.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Not applicable	Not applicable: This member-authored PR reports Blacksmith Testbox full-suite and focused-run results; the external-contributor proof gate does not apply.
Evidence reviewed	4 items	Current source reproduces the control gap: Current main only denies messaging tools; it does not set OpenClaw Code Mode or Tool Search, leaving the benchmark behavior dependent on the installed OpenClaw default. Retired inputs are not a current-main interface: Current main contains no references to either retired tool-mode input, so rejecting them in this PR does not break a shipped ShellBench configuration surface. PR implements explicit modes and export validation: The PR defaults unset runs to direct mode, writes explicit Code Mode or directory Tool Search configuration when selected, and validates exported root and child trajectories.
Findings	None	None.
Security	None	None.

How this fits together

ShellBench’s native evaluator creates run manifests, launches OpenClaw benchmark jobs, and converts exported session trajectories into scored evidence. This change controls the OpenClaw tool surface and rejects incomplete traces before they affect benchmark results.

flowchart LR
  A[Benchmark plan] --> B[Run manifest]
  B --> C[OpenClaw harness]
  C --> D[Session trajectory export]
  D --> E[Trace validation]
  E --> F[Benchmark scoring]

Before merge

Resolve merge risk (P1) - Unset-mode OpenClaw batches will change from the installed runtime default to explicit direct mode, so existing results should not be compared with post-merge direct/code results without fresh matched runs.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Production versus tests	production +699/-477, tests +1,116/-475	The large trace-pipeline change is accompanied by more focused test coverage than production growth.

Merge-risk options

Maintainer options:

Accept the benchmark baseline reset (recommended)
Treat existing unset-mode OpenClaw results as legacy data and require fresh matched direct and Code Mode batches after merge.

Technical review

Best possible solution:

Land the explicit control and public-export validation path, then establish fresh matched direct and Code Mode benchmark evidence while treating older unset-mode results as legacy evidence.

Do we have a high-confidence way to reproduce the issue?

Yes, from source: current main omits the relevant OpenClaw controls, while the proposed path explicitly writes and validates each selected mode. No fresh remote benchmark was run in this read-only review.

Is this the best way to solve the issue?

Yes; configuring the intended OpenClaw mode and using public trajectory exports is a narrower, more maintainable fix than relying on runtime defaults and private session reconstruction.

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label justifications:

P2: The PR repairs benchmark-evidence correctness without demonstrating an urgent production outage.
merge-risk: 🚨 compatibility: Unset benchmark runs change from an installed OpenClaw default to explicit direct mode, changing upgrade-time result semantics.
rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: This member-authored PR reports Blacksmith Testbox full-suite and focused-run results; the external-contributor proof gate does not apply.

Evidence

What I checked:

Current source reproduces the control gap: Current main only denies messaging tools; it does not set OpenClaw Code Mode or Tool Search, leaving the benchmark behavior dependent on the installed OpenClaw default. (scripts/native_eval/harnesses.py:583, 884dd1bb5511)
Retired inputs are not a current-main interface: Current main contains no references to either retired tool-mode input, so rejecting them in this PR does not break a shipped ShellBench configuration surface. (884dd1bb5511)
PR implements explicit modes and export validation: The PR defaults unset runs to direct mode, writes explicit Code Mode or directory Tool Search configuration when selected, and validates exported root and child trajectories. (scripts/native_eval/harnesses.py:367, f5b31cb1260b)
Feature provenance: Current main’s native-evaluation implementation most recently came from the delegated-trace reconstruction commit by the same area contributor. (scripts/native_eval/harnesses.py:527, 884dd1bb5511)

Likely related people:

vincentkoc: The current main history for the harness, trajectories, fleet controller, runner, and models is led by this contributor, including the latest merged trace reconstruction. (role: primary native-evaluation contributor and recent area author; confidence: high; commits: 69f75c6629c4, b9acd9f7a010, 884dd1bb5511; files: scripts/native_eval/harnesses.py, scripts/native_eval/harness_trajectories.py, scripts/native_eval/fleet.py)

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (28 earlier review cycles; latest 8 shown)

reviewed 2026-08-03T01:49:10.537Z sha f5b31cb :: found issues before merge. :: [P1] Preserve an upgrade path for retired tool-mode configuration | [P1] Translate legacy environment-based benchmark invocation
reviewed 2026-08-03T04:09:09.432Z sha f5b31cb :: found issues before merge. :: [P1] Preserve retired manifest tool-mode configuration | [P1] Translate the legacy environment tool-mode input
reviewed 2026-08-03T09:05:51.332Z sha f5b31cb :: needs changes before merge. :: [P1] Translate retired manifest tool-mode values | [P1] Translate the legacy environment tool-mode input
reviewed 2026-08-03T20:14:25.357Z sha f5b31cb :: needs changes before merge. :: [P1] Translate retired manifest tool-mode values | [P1] Translate the legacy environment tool-mode input
reviewed 2026-08-04T15:03:31.890Z sha f5b31cb :: needs changes before merge. :: [P1] Translate retired manifest tool-mode values | [P1] Translate the legacy environment tool-mode input
reviewed 2026-08-04T18:03:25.379Z sha f5b31cb :: found issues before merge. :: [P1] Translate retired manifest tool-mode values | [P1] Translate the legacy environment tool-mode input
reviewed 2026-08-04T21:04:25.672Z sha f5b31cb :: found issues before merge. :: [P1] Translate retired manifest tool-mode values | [P1] Translate the legacy environment tool-mode input
reviewed 2026-08-04T23:03:26.634Z sha f5b31cb :: needs changes before merge. :: [P1] Preserve retired manifest tool-mode values | [P1] Translate the retired environment tool-mode input

fix(native-eval): make tool search controls explicit

7f31256

vincentkoc marked this pull request as ready for review July 29, 2026 20:56

vincentkoc requested a review from a team as a code owner July 29, 2026 20:56

fix(native-eval): benchmark genuine OpenClaw code mode

ebe901e

vincentkoc changed the title ~~fix(native-eval): make OpenClaw Tool Search controls explicit~~ fix(native-eval): benchmark genuine OpenClaw code mode Jul 30, 2026

clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. labels Jul 30, 2026

test(native-eval): fix tool mode validation coverage

1f99e06

clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. labels Jul 30, 2026

vincentkoc added 2 commits July 30, 2026 06:12

fix(native-eval): validate public OpenClaw trajectories

7f6e2db

fix(native-eval): validate observed code-mode surface

f5b31cb

clawsweeper Bot mentioned this pull request Jul 30, 2026

fix(native-eval): make OpenClaw Tool Search controls explicit #61

Open

Navigation Menu

Uh oh!

Uh oh!

fix(native-eval): benchmark genuine OpenClaw code mode - #62

fix(native-eval): benchmark genuine OpenClaw code mode#62
vincentkoc wants to merge 5 commits into
mainfrom
fix/openclaw-tool-search-explicit-off

vincentkoc commented Jul 29, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jul 29, 2026 •

edited

Loading

Security

Review metrics

Merge-risk options

Technical review

Labels

Evidence

Rating scale

Workflow

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

vincentkoc commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why?

Changes

Tests

Uh oh!

clawsweeper Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ClawSweeper review

What this changes

Regression provenance

Merge readiness

Review scores

Verification

How this fits together

Before merge

Security

Review metrics

Merge-risk options

Technical review

Labels

Evidence

Rating scale

Workflow

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vincentkoc commented Jul 29, 2026 •

edited

Loading

clawsweeper Bot commented Jul 29, 2026 •

edited

Loading